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Objective 


•  Summarize  and  assess  results  of  SEP 
reviews  to  date 

-  SEP  represents  what  is,  not  what  should  be 

-  A  measure  of  how  well  the  revitalization  of  SE 
is  going 


State  of  the  Practice  vs  State  of  the  Art 
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❖ 


Overview 


•  Background  -  policy,  programs,  reviewers 

•  Review  results 

•  Implications 

•  Recommendations 


r  SE  Implementation  Hierarchy 


Apologies  to  Maslow 
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•  Policy  Memo 
-  Feb  04 

•  Draft  AFI 


MEMORANDUM  FOR:  SRI  DIS'J  RJ  li  IJTION 


■UUDJEC.T:  I'oLiey  Jin  S_v.->;ejiL>  Engineering  jji  Dun 


Application  of  rigorous  systems  tnainsirrbg  disci:ii:  m:  ptirunoLn’  lo  :Iik 
Dquinmcnt’s  aoility  to  meer  (tic  challenge  of  dove  Loping  and  maintaining  ;iW?dwl 
h nr. p?  rapa-r-iliiy.  This  is  especially  mis  as  wc  strive  w  in^afc  ii:  creasing]  v 
(.■(iTTipiRJi  3V£tefjrLs  in  a  family -uf-systems,  sysiffmsrf-sy  slams,  net-ccnbk-  vvarfare  uunlL’xr. 
SyMCLii-i  erigiTi <v-Lug  pio vines  the  integrm ini;  icchiiical  processus  lo  define  and  fiance- 
system  |,l:,  I....  jiiMte,  ajst:  scli^du]e:  andris*.  i:  miic  be  Linbcddird  ir  p'muijjif 

Jitld  piTtormiid  across  the  L-:i'.irc  acquit  hinn  life 

1  owjnd  insit  end,  [  jrn  coi s jJis.1 11: it;  iLc  loLJfiuinji  policy.  efffi i-- L vc  imn ’_e -rli ;iL c Ly  and 
to  Included  in  die  cll-xI  ravisioa  of  ihu  DoD  5000  semes  acquisition  diieiim^.s: 

Systems  huglneeriiijr  JShl.  All  programs  responding  go  a  m  pain  li  lbs  nr 
-cqiLrecienLs  divimiejit,  regardLcas  etii/quisiriOL’.  ctiegory.  shall  apply  n 
iuIiiim  SE  approach.  Thai  balances  inisi  yysiem  paforaiiitiLe  Lind  total  ' 

UMCLihip  costs  ifcitair.  the  family-of-syslarcs.  systems-tita  sterns  cmrc.Hi. 

L'lCunirris  shi-ll  develop  n  Systems  Fin^Ticcjiiig  Flan  [Sk\y;  +'n:  VTiKmr 
Hi  ‘j;  i]  on  A  '-Hi  i  v:  l  y  i.MJM  l  appro’.-  n  I  Ltt  e-  m  junrao  n  ■*  il  h  cock  M  LI  „-s  Lgj  Jf 
review,  s;nd  integuied  with  die  Aciiniuiion  Strategy.  This  plan  shall  describe 
He  proaj'im’s  overall  Kclmivul  uppioceh:  iueludlng  pmi.r:^,  issuama* 

LiieL:  ics.  Hud  applicable  performance  incimii^cs.  li  sliul  I  also  detail  ihi:  iinunjj. 
conduct..  and  success  criteria  of  technical  reviews. 

In  si:pport  of  the  ::hcive  policy,  tin:  Director,  Dtifensc  Systems 

■l  Wcr.:ify  die  i^itiircinent  iWa  SEF  in  DODI  dOCJCI.2,  lijkI  provide  specific 
i  L-nLent  guidance  tailoraiile  by  I  hr  ML1A  :ji  the  Dere:is?  Acquisition  Guide  In  ml: 

Lv  Assess  i he  adequacy  of  current  U-epdrlmei-.L-leveL  SE  related  policies, 
processes,  prjicTiivs.  guidance.  tools.  l::;J  education  and  training  arid  rruunvrie-d 
"o  me  necessary  changes. 
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About  the  Programs 


❖ 


•  Non-Space  AF  programs  at  a  milestone 

-  Small  #  of  programs 

•  Numerous  other  programs 

-  Starting  SEPs 
-Asking  questions 

-  Quick  reviews 
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Air  Force  Reviewers 


❖ 


•  SAF/ACE  and  AQR 

•  Extended  Staff 

-  AFMC/EN 


-  AF  Center  for  Systems  Engineering 
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Areas  Studied 


❖ 


•  Requirements  definition 

•  Processes 

•  Risk 

•  Key  Performance  Parameters 

•  Enterprise  SE 

•  Multiple  Reviews 

•  Authorship 

•  SEP  Size 


4  i 


Requirement  Definition 


❖ 


-60%  of  programs  adequately  defined 
their  requirements  below  the  ICD/CDD 

-Many  programs  can  point  to  a  “textbook” 
requirements  analysis/decomposition 
process 

-Fewer  can  point  to  a  configuration 
controlled  specification 


Quotes 

“We  don’t  have  any  requirements” 
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Processes 


❖ 


•  27%  of  programs  described  processes 

•  The  rest  either 

-  Don’t  have  a  process 

-  Don’t  know  the  process 


Process  101 

If  you  can’t  document  the  process 

You  don’t  have  one 

•  55%  of  programs  defined  their  risks 

•  The  rest 

-  Simply  don’t  know  what  the  risks  are 

-  Not  a  integral  part  of  the  program 


Quotes 

“I  can’t  list  my  risks  in  the  SEP.  They  change  daily” 

“Why  do  you  need  to  know  what  the  program  risks  are  to  do  SE  planning” 
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Performance  Parameters 


•  73%  of  the  programs  list  KPPs 

-  KPPs  are  clearly  stated  as  a  SEP  requirement 


Quotes 

“What  have  KPPs  got  to  do  with  SE?” 

“I  can’t  list  all  of  my  program’s  KPPs  in  the  SEP.  We  have  hundreds” 

“I  don’t  have  any  KPPs” 
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Scheduled  Design  Reviews 


❖ 


•  55%  of  programs  have  entry  and  exit 
criteria  for  design  reviews 


“  We  are  not  there  yet” 


Quotes 
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Integrated  SE  Processes 


❖ 


•  10-20%  of  programs  have  fully 
integrated  SE  processes 

-  Risk 

-  Design  reviews 

-  Configuration  Management 
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Multiple  reviews 


❖ 


•  Few  programs  approved  without  multiple 
iterations 

•  Approvals  with  comments 
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Authorship 


❖ 


•  SEPs  written  by 

-  Prime  contractor 

-  Task  order  contractor 

-  Reserve  Officer 

-  Junior  members  of  program 
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Big  SEPs 


•  Don’t  know  what  “they”  want 

•  Don’t  know  what  my  program  is  doing 

•  Give  “them”  lots  of  stuff  and  hope  they 
stumble  over  what  they  want 

•  Tutorial 
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State  of  the  Practice 


❖ 


•  Not  @  100%  in  any  of  the  areas  reviewed 

-  Requirements  definition 

-  Processes 

-  Design  reviews 

•  Shortfall  is  in  SE  fundamentals 


State  of  the  practice  well  below  the  state  of  the  art 


SE  Implementation  Hierarchy 


Systems/SoS  Level  Optimization 
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Results  -  Possible  Factors 


❖ 


•  Requirement  to  document  SE  planning 
a  SEP  is  new 

•  Format  confusion 

-  What  do  they  really  want? 

•  Years  of  negative  learning 
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Atrophied  SE  Talent 


❖ 


•  AFMC  has  half  the  number  of  engineers  as 
in  the  early  80s 

•  Engineers  hired  in  the  last  decade+  were 
trained  in  a  less  disciplined  SE  environment 

•  SE  talent  still  exists  in  AF/center 

-  Generally  at  a  higher  level 

•  That  limited  talent  is  probably  not  working 
on  the  program 

Consultants  can  only  do  so  much 
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Implications  -  More  Help 


❖ 


•  SEPs  indicate  continuing,  significant 
problems  with  the  implementation  of  SE 

•  The  powers  that  be  will  “Inspect  in  good 
SE” 

-Wing,  Group,  Squadron,  PEO/Center,  SAF/AQ 


What  gets  inspected  gets  improved 
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\^lmplications-SEP  as  Audit  Toof 


•  More  status  will  be  required  in  SEPs 

•  Approval  with  comments 
-  Update  in  90  -  120  days 
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Recommendation 


❖ 


•  Continue  this  type  of  analysis 

-Across  programs  and  overtime 

•  Develop  PEO  checklists 

-  Start  with  OSD  SEP  checklist 

-  Tailored/specific  to  product  line 

•  Require  just-in-time  training 

-  Event/milestone 


